Bounds for the Tracking Error of First-Order Online Optimization Methods

This paper investigates online algorithms for smooth time-varying optimization problems, focusing first on methods with constant step-size, momentum, and extrapolation-length. Assuming strong convexity, precise results the tracking iterate error (the limit supremum of norm difference between optimal solution iterates) gradient descent are derived. The then considers a general first-order framework, where universal lower bound is established. Furthermore, method using “long-steps” proposed shown to achieve up fixed constant. compared specific examples. Finally, analyzes effect regularization when cost not strongly convex. With regularization, it possible non-regret bound. ends by testing accelerated regularized synthetic least-squares logistic regression respectively.